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Method for Producing Cloud Free, and Cloud-Shadow Free, Images 
Field of the Invention 

This invention relates to a method for producing cloud free, and cloud-shadow free, 
images and refers particularly, though not exclusively, to such a method for producing 



Background to the Invention 

It is well known that optical remote sensing images often encounter the problem of cloud 
cover, either partial or complete, especially over humid, tropical regions. There is also 
the problem of cloud shadow. In the past there have been many attempts to eliminate the 
problem of clouds appearing in images of a region, the images being taken using optical 
remote sensing. 

The conventional method for generating a cloud free mosaic is by removing the clouds. 
In undertaking this process, an image containing the least cloud cover is taken as the base 
image. The cloudy areas in the image are masked out, and then filled in by cloud-free 
areas from other images acquired at different times This is no more than a manual "cut- 
and-paste" method. 

There have been attempts to automate the procedure. The most common way is to 
employ a simple intensity threshold process to discriminate the bright cloudy areas and 
dark cloud shadows from non-cloud areas. This method cannot handle thin clouds and 
cloud shadows, and often confuse bright land surfaces as clouds. There have been very 
few proposals for eluninating cloud shadows. 



such images from remote sensing using optical sensors« 



Consideration of Prior Art 



One propojal for automating the process is disclosed in US patent specification 
6,233,369. This discloses a system that incorporates a mask for the purpose of 
perfoxnoing morphological image processing on one or more adjacent pixels in which a 
mask is inci>rporated into a binary image by processing image data which are encoded 
using 2 bits rather than the usual 1 bit The specification is directed at the edges of the 
image where each pixel may not have a complete compliment of neighbours. In this way 
the second bit is a mask enable bit that directs the processing engine to pass the original 
data througl-i to the output image regardless of the processing result for that pixel. This 
allows the masked pixel data is permitted to participate in the computation of all its 
neighbouring pixel' s results. 

In US patent specification 5,612,901 there is disclosed an apparatus and method for 
cloud masking in an image of a body of water. It extracts cloud edge information 
through local segmentation of the image and discriminates between cloud free and cloud 
contaminated pixels on the basis that clouds are brighter and colder than the surrounding 
ocean. The cloud-contaminated pixels are then removed. 

The disclosure of the specification of US patent 5,923,383 is directed at an image 
enhancemeni: method using histogram equalisation so that the brighmess of an image is 
not significajitly changed, and the noise is not amplified. This is achieved by expressing 
the input im^^ge in a predetermined gray levels by calculating the distribution of the gray 
levels of the input image while constraining the number of occurrences of each gray 
level to be within a predeteimined value, and then performing histogram equalisation on 
the input image based on the calculated distribution of gray levels obtained previously. 

On a similar basis, the disclosure of EP 0366099 is directed at a method of image 
enhancement through the modification of the image histogram by using two matrixes 

In EP 0504876A2 there is disclosed a method and apparatus for enhancing an image by 
further processing in an independent maimer the non-brightness information in the 
image. 



Japanese 10-063836 relates to a method for the highlighting of the image using a 
morphological operation. 



In the paper titled "Improved "Cloud-Free Multi-Scene Mosaics of Spot Images'* by the 
present inveiators and Lim, Hok (Proceedings of the 19* Asian Conference on Remote 
Sensing, 1999) there is disclosed an algorithm for automatic generation of "cloud-free" 
scenes from multiple, multi*spectral images viithin a specified time interval over a given 
region. By creating a mosaic using the cloud-free areas in the set of multi-spectral 
images, a reasonably cloud-frec composite image can be made. The algorithm disclosed 
in the paper does not address the problem of creating a cloud-frce mosaic from multiple 
panchromatic images. 

The inputs to the system are multispectral images of the same region acquired vvithin a 
specified time interval, pre-processed to level 2A or 2B. The images are also co- 
registered before being fed into the system. The sensor captures data in three spectral 
bands: the green band, red band, and near-infrared band. The radiometric balancing 
procedure only makes a correction for dififerences in sensor gains, solar incidence angles 
and solar flax between the acquired scenes and no attempt is made to correct for 
atmospheric effects. 

After radiometric balancing, the brightness of pixels at the same location from two 
different scenes will be a litde different due to the atmospheric effects, especially in low- 
albedo vegetated areas. The pre-processing procedure tries to make a balance between 
the scenes for the differences caused mainly by atmospheric effects. After radiometric 
balancing, one image from the set of images is chosen as the reference image. For each 
band, the pixel values of all other images in the same set are adjusted. 

The pixel ranking procedure uses the pixel intensity and suitably chosen band ratios to 
rank the pixels in order of "cloudiness" and "shadowness" according to predefined 
ranking criteria. 

A shadow intensity threshold and a cloud intensity threshold are determined from the 
intensity histogram. The pixel ranking procedure uses these shadow and cloud thresholds 
to rank the pixels in order of "cloudiness" and "shadowness". Each of the non-cloud and 
non-shadow pixels in the images is classified into one of four broad classes based on the 
band ratios: vegetation, building, water and others. 



Pixels with lower rank values are more superior and are more likely to be selected. 
Pixels with intensities falling between the* shadow and cloud thresholds are the most 
superior, and are regarded as the "good pixels". Where no good pixels arc available, the 
"shadow pi:cels" are preferred over the "cloud pixels". Where all pixels at a given 
location are "shadow pixels*', the brightest shadow pixels will be chosen. In locations 
where all pixels have been classified as "cloud pixels", the darkest cloud pixels will be 
selected. 

The rank*! and rank-2 index maps are used to merge the multi-scenes firom the same set 
of images. If the pixel at a given location has been classified as "vegetation pixel", the 
pixels from die rank-1 image and the rank-2 image at that location are averaged together 
in order to avoid sudden spatial discontinuities in the final mosaic image. Otherwise, the 
pixels from ihe rank-1 image are used. 

As many pixels as possible in the neighbourhood of a given location come from the 
same scene. The image that is deemed to have the lowest cloud coverage by visual 
inspection is chosen to be the base image. Cloud and shadow thresholds are then applied 
to this base image to delineate the cloud shadows and the cloud covered areas. In the 
next step of mosaic generation, only the delineated cloud and shadow areas will be 
replaced witli pixels from the merged image generated from the previous step. 

The final mosaic is composed from the merged images and the base image. These 
images are geo-referenced to a base map using control points. The mosaic generation 
transforms tlie coordinates of the pixels in the merged images' and the base image into 
map coordinates and put the pixels onto the final image map. 

Cloud masking methods based on intensity thresholds cannot handle thin clouds and 
cloud shadows. They often confuse bright land surfaces as clouds, and dark land 
surfaces as shadows. In multi-spectral images with two or more spectral bands, the 
spectral, or colour, information can be used to discriminate different land cover types 
from clouds. However, in panchromatic or grey scale images, the colour information is 
absent, and it is even more difficult to discriminate bright land surfaces from clouds, and 
dark land surlfaces from cloud shadows. 



It is therefore the principal object of the present invention to address their problems. 

A further object is to provide a method for producing cloud free, and cloud-shadow free, 
images from cloudy panchromatic or grey scale images. 

A final object of the present invention is to provide cloud free, and cloud-shadow free, 
images from cloudy multi-spectral images. 

Summary of the Inveutioa 

The present invention employs pixel ranking. In addition to generating cloud and shadow 
masks by clcissifying a group of pixels as cloud, shadow, or noncloud-nonshadow. Each 
pixel in each of the images may be ranked according to predefined ranking criteria, and 
the highest ranked pixels are preferably used to compose the mosaic. 

By using size and shape information of the bright pixel clusters it is possible to 
discriminate bright land surface and buildings from clouds. It is also possible to predict 
the approximate locations of cloud shadows based on the knowledge of solar illumination 
direction, sensor viewing direction and typical cloud heights. 

The present i nvention also provides for the use of intensity gradients to enable automatic 
searching for the locations of cloud shadows near the edges of clouds. 

The present invention also provides for applying a morphological filter to the cloud 
masks detected by use of an intensity threshold process in order to include thin clouds 
around the edges of thick clouds. 

The present invention also provides for using a conditional majority filter in addition to 
the ranking criteria to include as large a patch of neighbouring "good pixels" as possible 



in the generation of the mosaic. The merging of rank- 1 and rank-2 pixels under certain 
conditions niay produce a more pleasing visual, effect. 

If multiple images acquired at different time over a given region are available, it is 
practicable lo generate a reasonably cloud free composite scene by creating a mosaic of 
the cloud free areas in the set of images, assuming that the land covers do not change 
within the time interval. This is particularly relevant for composing "cloud-free" multi- 
scene mosaics of panchromatic and/or multi-spectral satellite images. 

The highest raking pixels may be considered as good pixels and the lowest ranking 
pixels are considered as bad pixels. The good pixels are preferably fimher classified into 
vegetation pixels and building pixels. The building pixels may include land clearings. 
The classification may depend on whether the pixel intensity is below or above a 
threshold for vegetation pixels. Darker good pixels may be preferred over brighter good 
pixels. 

The present invention also provides a cloud free and cloud-shadow free image produced 
by the above method. 

In a fmal fonn, the present invention provides a computer usable medium having a 
computer prc»gram code which is configured to cause a processor to execute one or more 
functions to enable the method described above to be performed on at least one 
computer. 

Description of the Drawing 

In order that rJie invention may be fully understood and readily be put into practical 
effect, there shall now be described by way of non-limitative example only a preferred 
embodiment of the present invention, the description being with reference to the 
accompanying illustrative drawing which is a schematic flow chart of the preferred 
method of the present invention. 

Description of Preferred Embodiment 



The inputs 1 to the system are a plural number of panchromatic and/or multi-spectral 
images of the same region acquired wittua a specified time interval, and that are co- 
registered. 

The images are subjected to two different processing streams. In the first stream, along 
the top of thie drawing, at 2 an intensity threshold method is mitially applied to generate 
a cloud ma;slc, and a cloud shadow mask, for each image. Confusion may arise when 
bright pixels of open land surfaces or buildings are mistaken as cloiid pixels. Such 
confusion may be resolved by making use of size and shape information of the bright 
pbcel clusters detected during the by threshold step. Clouds that need to be inasked are 
much larger than individual buildings. Man-made features such as buildings and land 
clearings nomrially have simple geometrical shapes. 

At 3, the size of the bright patches is calculated, and the lines and simple shapes of such 
things as t-uildings are detected. The intensity threshold method does not work 
adequately in generating cloud shadow masks. By using geometric modeling, as well as 
intensity gr:idients to automatically search for cloud shadows near cloud edges, the 
preferred Kiethod of the present invention compensates for the patch identified 
improperly in the automatic mask method. Furthermore, solar illumination direction, 
sensor viewing direction, and typical cloud heights infonnation may be used to predict 
the likely location of cloud shadows. This is of particular relevance once the locations of 
the clouds is determined. 

As there may be an intensity gradient at cloud edges, a fixed threshold method is used at 
step 4 to label any thin clouds at cloud edges, as non-cloud pixels. A morphological 
filter is used to dilate the cloud mask patch. The gray level is then balanced at 8 to 
compensate for differences caxised mainly by atmospheric effects. 

After constaicting the cloud mask and cloud shadow mask for each component image, in 
the second stream, iat 5 the gray levels are balanced; again to compensate for differences 
caused mainly by atmospheric effects. 
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The pixel nmking procedure at 9 uses the shadow, cloud thresholds, and ranking criteria 
described below, to rank the pixels in order of "cloudiness" and "shadowness". The pixel 
ranking proi:edure uses the pixel intensity to rank the pixels in order of "cloudiness" and 
"shadowness" according to predefined ranking criteria. 

hi this procedure, a shadow intensity threshold T^^ a vegetation intensity threshold 
and a cloud intensity threshold are determined from the intensity histogram. The pixel 
ranking prot:edure uses these shadow, vegetation and cloud thresholds to rank the pixels 
in order of "cloudiness" and "shadowness". Each of the non-cloud and non-shadow 
pixels in the images is classified into one of two broad classes based on the intensity: 
vegetation and building. 

For each inriage n from the set of N acquired images, each pixel at a location is 
assigned a rank based on the pixel intensity YnO.j) according to the following 

rules: 

(i) For Ts < (Yrn, Y^) < Ty, if Yj,^ < Y^ (class = "vegetation"), then < r„; 

(ii) For rv < iJm. ^ 7 c> if >m < (class « "building"), then < r^- 

(iii) If 7;,, < Ts and Y^ > Tq. then ryy^<ry(, 

(iv) For Yn < Ts, ifYrr, > r„, then r^^ < rn\ 
(V) For Ym, Y„ > T^. if Yjn < then < r„; 

In this scheme, pixels with lower rank values of r^^ are more superior and are more likely 
to be selected. Pixels with intensities falling between the shadow and cloud thresholds 
are the mosit superior, and are regarded as the "good pixels". The "good pixels" are 
further classified into ^Vegetation pixels" or "building pixels** (that also include land 
clearings) depending on whether the pixel intensity is below or above the vegetation 
threshold. The darker "good pixels" are preferred over the brighter "good pixels" as the 
brighter "good pixels" may be contaminated by thin clouds. Where no good pixels are 
available, the "shadow pixels" are prefenred over the "cloud pixels". Where all pixels at a 
given location are "shadow pixels", the brightest shadow pixels will be chosen. In 



locations wliere all pixels have been classified as "cloud pixels", the darkest cloud pixels 
will be sele(:ted. 

After ranking the pixels, the rank-r index map UriUj) representing the index n of the 
image with rank r at the pixel location can be generated at 10. It is preferred that 
only the rank-1 and rank-2 index maps are generated and kept for use in generating the 
cloud-&ee mosaics. 

In order to obtain improved visual effects, it is desirable to have as many pixels as 
possible in the neighborhood of a given location to come from the same image. A 
conditional majority filter procedure is applied to provide this. 

In the merging of sub-images at 6, the conditional majority filtered ranking index is used 
to merge the input multi-scenes that have been processed by the gray-level balance. 
Using the images v.ith cloud, cloud shadow masks and the merged image generated from 
the merging of sub-images procedure, the final cloud-free mosaic is composed at 7, The 
images resulting from the itnosaic process are co-registered with the map. The mosaic 
generation procedure will put the imsige from the mosaic process into the map at 1 1. 

When merging sub-images, the rank- 1 and rank-2 index maps are used to merge the 
multiple scenes from the same set of images. If the pixel at a given location has been 
classified as "vegetation pixel", the pixels from the rank-l image and the rank-2 image at 
that location are averaged together in order to avoid spatial discontinuities in the final 
mosaic image. Otherwise, the pixels from the rank-l image are used. 

The present invention also provides a computer readable medium such as a CDROM, 
disk, tape or the like, having a computer program thereon, the computer program being 
configured to cause a processor in a computer to execute one or more functions to enable 
to computer to perform the method as described above. 

The present invention also provides a computer usable medium having a computer 
program code which is configured to cause a processor to execute one or more functions 
to enable the method described above to be performed on at least one computer. 
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Whilst there has been described in the foregoing description a prefened embodiment of 
the present invention, it will be understood by those skilled in the technology that many 
variations ox modifications in the method of the present invention may be made without 
departing from the present invention. 
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The Claimit 

1 . A method for generating a cloud free and cloud-shadow fee image from a plurality of 
images of a region, the method including the steps of: 

(a) lanking pixels in order of cloudiness and shadowness; 

(b) {venerating cloud and shadow masks by classifying a group of pixels as cloud, 
shadow, or noncloud-honshadow; and 

(c) coating a mosaic from the plurality of images to form the cloud free and 
ciloud-shadow free image. 

2 . A method as claimed in claim 1, wherein each pixel in each of the images is ranked 
according to predefined ranking criteria, and the highest ranked pixels are used to 
compose the mosaic. 

3. A method as claimed in claim 1 or claim 2, wherein size and shape information of 
bright pixel clusters are used to discriminate any bright land surfaces and buildings 
from clouds. 

4. A method as claimed in any one of claims 1 to 3, wherein solar illumination 
direction, sensor viewing direction and typical cloud heights information is used to 
predict Likely locations of cloud shadows. 

5. A methcd as claimed in any one of claims 1 to 4, wherein intensity gradients are 
used to search for locations of cloud shadows near cloud edges. 

6. A method as claimed in claim 5, fiuther including the step of applying a 
moiphological filter to the cloud masks detected by the intensity gradients to locate 
and include thin clouds around the edges of thick clouds. 

7. A method as claimed in any one of claims 1 to 6, including the step of using a 
conditional majority filter in addition to the ranking criteria to include as large a 
patch of aeighbouring good pixels as possible in the generation of the mosaic. 
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8. A methcid as claimed in any one of claims 1 to 7, wherein the plurality of images is 
panchromatic satellite images. 

9. A method as claimed in any one of claims 1 to 7, wherein the plurality of images is 
multi*spectral images. 

10. A method as claimed in any one of claims 1 to 9, wherein the highest raking pixels 
are considered as good pixels and the lowest ranking pixels are considered as bad 
pixels. 

1 1 . A method as claimed in claim 10, wherein the good pixels are fiuther classified into 
vegetation pixels and building pixels. 

12. A method as claimed in claim 1 1, wherein the building pixels include land clearings. 

13. A method as claimed in claim 11 or claim 12, wherein the classification depends on 
whether die pixel intensity is below or above a threshold for vegetation pixels. 

14. A method as claimed in any one of claims 10 to 13, wherein darker good pixels are 
preferred over brighter good pixels. 

15. A cloud free and cloud-shadow free image produced by the method of any one of 
claims 1 to 14. 

16. A computer usable medium having a computer program code which is configured to 
cause a processor to execute one or more steps to enable a computer to perform the 
method cf any one of claims 1 to 14. 
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ABSTRACT 



Method for Producing a Cloud Free and Cloud-Shadow Free Image 



10 A method for generating a cloud free and cloud-shadow free image from a plurality of 
images of a region, the method including the steps of ranking pixels in order of 
cloudiness and shadowness, generating cloud and shadow masks by classifying a group 
of pixels as cloud, shadow, or noncloud-nonshadow, and creating a mosaic &om the 
plurality of images to form the cloud free and cloud-shadow free hnage. 
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Figure 1 
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